On homoclinic and heteroclinic orbits of Chen's System

The classical Lorenz system is a three-dimensional autonomous dynamical system with only two quadratic terms but displays very complex dynamical behaviors (see [Guckenheimer & Holmes, 1983; Lorenz, 1963]). With a similar structure, the recently discovered Chen’s system is of particular interest because of its complex topological structure and its duality to the Lorenz system (see [Chen & Ueta, 1999; Ueta & Chen, 2000]). Some fundamental dynamical properties of Chen’s system have been carefully studied from a mathematical point of view (see for example [Li et al., 2004; Zhou et al., 2004; Agiza & Yassen, 2001; C̆elikovský & Chen, 2002, 2005; Lu et al., 2002; Lü et al., 2002a, 2002b; Yassen, 2003; Yu & Xie, 2001] and the references cited there). Due to the extremely complex structure of its attractor, many properties of Chen’s system remain to be further investigated. Homoclinic orbits and heteroclinic orbits are important concepts in the study of bifurcation of vector fields and chaos. Many chaotic behaviors of a complex system are related to the existence or nonexistence of these kinds of orbits in the system. In the present paper, we are concerned with these properties of Chen’s system. Recall that a homoclinic orbit is a trajectory that is doubly asymptotic to an equilibrium point, or is a closed orbit asymptotic to itself. A heteroclinic orbit is a trajectory that connects an equilibrium point or a closed orbit to another equilibrium point or another closed orbit, respectively. Recall also that Chen’s system is described by the following three-dimensional smooth quadratic autonomous system:   ẋ = a(y − x) ẏ = (c− a)x− xz + cy ż = xy − bz, (1)